Augmented Reality Methods and Systems Including Optical Merging of a Plurality of Component Optical Images

ABSTRACT

Embodiments of the present disclosure provide augmented reality methods and systems where two or more component optical images are optically overlaid via the use of one or more beam splitters to become composite optical images wherein in some embodiments a second component optical image is an electronic optical image (an image from an electronically controlled emission source) while the first component optical image is one of a physical optical image (an image of a physical object from which diffuse reflection occurs), an electronic optical image, an emission optical image (an image from a non-electronic source that emits radiation), or a hybrid optical image (which composed of at least two of a physical optical image, and electronic optical image, or an emission optical image). In some embodiments the first and second component optical images are used to provide feedback concerning the quality of the overlaying and appropriate correction factors to improve the overlay quality.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/356,150 filed Jun. 18, 2010. This referencedapplication is incorporated herein by reference as if set forth in fullherein.

FIELD OF THE INVENTION

The present invention relates generally to the field of imaging, moreparticularly to the field of augmented reality where two or more imagesare overlaid and viewed together by an observer, and even moreparticularly to such systems and methods wherein the merging of at leasttwo component optical images into a composite optical image which are tobe viewed together occurs optically. In the various embodiments a secondcomponent optical image is an electronic optical image (an image from anelectronically controlled emission source) while the first componentoptical image is one of a physical optical image (an image of a physicalobject from which diffuse reflection occurs), an electronic opticalimage, an emission optical image (an image from a non-electronic sourcethat emits radiation), or a hybrid optical image (which composed of atleast two of a physical optical image, and electronic optical image, oran emission optical image).

BACKGROUND OF THE INVENTION

Augmented reality (AR) is a term that is applied to composite opticalimages and methods and systems for producing those composite opticalimages wherein the composite optical images are typically composed ofcomponent optical images, one of which is typically an image of aphysical or real object or scene while another component optical imageis computer or electronically created, placed, modified, or manipulatedto provide an enhanced or alternative understanding of the physicalobject or scene.

A need exists in AR methods and systems for forming composite opticalimages using optical merging of component optical images withimprovements in overlaying of the optical component optical images.

SUMMARY OF THE INVENTION

It is an object of some embodiments of the invention to provide animproved AR system and methods for use in medical applications (e.g.tissue and tool viewing during surgery, performance of diagnosticprocedures, interpretation of the results of diagnostic procedures, andthe like) in manufacturing applications (e.g. product design evaluationand verification, inspection during component manufacturing, componentalignment during assembly, inspection after assembly, and the like)and/or in research applications (e.g. data gathering and presentation,and the like).

It is an object of some embodiments of the invention to provide improvedAR systems and methods that provide improved overlaying (i.e. merging)of component optical images when producing a composite optical image.

It is an object of some embodiments of the invention to provide improvedAR systems and methods that use optical overlaying for viewing mergedcomponent optical images (i.e. composite optical images) of first andsecond component optical images wherein the second component opticalimage is an electronic optical image and the first component opticalimage is one of a physical optical image, an electronic optical image,an emission optical image, or a hybrid optical image wherein electroniccapture of updated first images are used in determining updatedpositioning and orientation information for overlaying the secondcomponent optical images with the first component optical images.

It is an object of some embodiment of the invention to provide improvedAR systems and methods that use optical overlaying for viewing aplurality of component optical images along with electronic capture of aplurality of component optical images to provide feedback and ability toprogressively adjust and minimize overlay errors.

Other objects and advantages of various embodiments of the inventionwill be apparent to those of skill in the art upon review of theteachings herein. The various embodiments of the invention, set forthexplicitly herein or otherwise ascertained from the teachings herein,may address one or more of the above objects alone or in combination, oralternatively may address some other object ascertained from theteachings herein. It is not necessarily intended that all objects beaddressed by any single aspect of the invention even though that may bethe case with regard to some aspects.

Various terms as used herein may have meanings that are narrowed,enhanced, or simply different from their normal meanings and such termsare explicitly or inherently defined herein, e.g. in the detaileddescription section to follow.

A first aspect of the invention provides a method for providingcomposite optical images to an eye or eyes of an observer wherein thecomposite optical images include first component optical images of anobject or source along with second component optical images that are tobe displayed in relationship to the first component optical images, themethod includes: (a) directing first component optical images along afirst optical path to a first beam splitter; (b) directing the firstcomponent optical images along second and third optical paths from thefirst beam splitter to the eye or eyes of the observer and to a firstimage capture device, respectively; (c) producing first electronic datacorresponding to the first component optical images captured by thefirst image capture device and using said first electronic data incombination with other data to provide second electronic data; (d)providing said second electronic data to an electronic image displaydevice to provide second component optical images which are directedalong a fourth optical path onto a second beam splitter; (e) directingthe second component optical images along at least a fifth optical paththat extends from the second beam splitter to the eye or eyes of theobserver wherein the fifth optical path and second optical path at leastpartially overlap.

Numerous variations of the first aspect of the invention exist andinclude, for example: (1) the second beam splitter being the same as thefirst beam splitter and the second and fifth optical paths being thesame optical path; (2) a sixth optical path extending from the secondbeam splitter to a second optical image capture device and whereincaptured second component optical image information provides at least aportion of the other data and is analyzed in combination with the firstelectronic data to produce said second electronic data, wherein thesecond image capture device is the same as the first image capturedevice; (3) a sixth optical path extending from the second beam splitterto a second optical image capture device and wherein captured secondcomponent optical image information provides at least a portion of theother data and is analyzed in combination with the first electronic datato produce said second electronic data, wherein the second image capturedevice is different from the first image capture device; (4) the firstcomponent optical images include optical images selected from the groupconsisting of (a) physical optical images, (b) electronic opticalimages, (c) emission optical images, and (4) hybrid optical images; (5)the second beam splitter is different from the first beam splitter; (6)the second beam splitter is different from the first beam splitter and athird beam splitter is provided that is different from both the firstand second beam splitters and which is located along the fourth opticalpath which splits the fourth optical path into one branch that continuesto the second beam splitter and one branch that directs the secondcomponent optical image onto a second image capture device which in turnprovides at least portion of the other data and is analyzed incombination with the first electronic data to produce said secondelectronic data; (7) the first component optical images include physicaloptical images and wherein the method additionally includes directinglight from a light source onto an object located at the beginning of thefirst optical path that gives rise to the physical optical images; (8)the first component optical images include physical optical images andwherein the method additionally includes directing light from a lightsource onto an object located at the beginning of the first optical paththat gives rise to the physical optical images, and wherein the lightform the light source includes light selected from the group consistingof (a) visible light, (b) UV light, and (c) IR light; (9) the firstcomponent optical images include physical optical images and wherein themethod additionally includes directing light from a light source onto anobject located at the beginning of the first optical path that givesrise to the physical optical images; and one or more optical componentsare provided between the light source and the object wherein at leastone of the one or more optical components is selected from the groupconsisting of: (a) a lens, (b) a mirror, (c) a mechanical or electronicshutter, (d) a prism, (e) a diffraction grating, and (f) a filter; (10)providing one or more optical components located along one or more ofthe first-fifth optical paths wherein the one or more optical componentsincludes at least one component selected from the group consisting of:(1) a lens, (2) a mirror, (3) a shutter, (4) a prism, (5) a diffractiongrating, and (6) a filter; (11) the first and second beam splitters arethe same beam splitter and the images reaching the beam splitter via thefirst and fourth optical paths are provided in time modulated forms suchthat images reaching the image capture device along the third opticalpath at selected times are the images carried along the first opticalpath but not the fourth optical path while at other times the imagesreaching the image capture device are the images carried along thefourth optical path but not the first optical path; (12) the imagesrepresented by the data provided along the second electronic path anddisplayed by the electronic image display device do not include imagesprovided along the first optical path; (13) the using of the firstelectronic data includes determining positioning information that placesthe second component optical images and the first component opticalimages within a desired tolerance by calculating a display position forthe second component optical images based, least in part, on capturedpositions of the first component optical images; (14) the using of thefirst electronic data includes determining positioning information thatplaces the second component optical images and the first componentoptical images within a desired tolerance by calculating a displayposition for the second component optical images based, least in part,on captured positions of the first component optical images and onspatial comparisons of prior captured first component optical images andprior captured second component optical images; (15) adjusting anoptical path by adjusting at least one component of the of system,wherein the component is selected from the group consisting of (a) atleast one beam splitter, (b) the display device, (c) at least one imagecapture device, (d) a positioning of the object, (e) an image focusingcomponent located along one of the optical paths, (f) a filter locatedalong one of the optical paths, (g) a shutter located along one of theoptical paths, (h) an aperture located along one of the optical paths,and (i) a mirror located along one of the optical paths; (16) adjustingan optical path by adjusting at least one component of the of system (a)manually during operation, (b) manually during calibration, (c)partially automatically during operation, (d) partially automaticallyduring calibration, (e) automatically during operation, and/or (f)automatically during calibration; (17) automatically adjusting anoptical path by adjusting at least one component of the of system using,at least in part, feedback provided by second component images capturedby a second image capture device; (18) the using includes processing thefirst electronic data using a programmed computer that is alsoprogrammed to control one or more system components and wherein at leastone of the one more system components are selected from: (a) anelectronic shutter located along one of the optical paths, (b) a lightsource that irradiates the object and is capable of output modulation,(c) am image capture device, (d) the display device, (e) a focusingdevice located along one of the optical paths, (f) at least one stagecoupled to at least one component for controlling one or more of aposition and an orientation of the component, (g) an electronicallycontrolled aperture located along one of the optical paths, and (h) anelectronically controlled filter located along at least one of theoptical paths; (19) the composite optical images are provided during aprocedure selected from the group consisting of (a) a medical procedure,(b) a medical ophthalmic procedure, (c) an ophthalmic intraocular lensreplacement procedure, (d) an ophthalmic phacoemulsification procedure,(e) a medical restorative procedure, (f) a medical therapeuticprocedure, (g) a medical diagnostic procedure, (h) a medicalpreventative procedure, (i) a medical research procedure, (j) amanufacturing inspection procedure, (k) a manufacturing assemblyprocedure, and (l) a research or engineering visualization ordemonstration procedure; or (20) the object or source includes an objectselected from the group consisting of (a) human tissue, (b) a humanorgan, (c) a human eye, (d) a rigid object, (e) a deformable object, (f)an expandable or contractible object and (g) an animate object, (h) aninanimate object; (21) the second component optical images include oneor more images selected from the group consisting of (a) capturedphysical object images, (b) magnetic resonance images, (c) coherencetomography image(s), (d) optical coherence tomography images, (e) acomputer generated polygonal graphic representation of an object, (f) animage that is in a format that can be manipulated by a graphicalanimation program, (g) an image that is in a format the can bemanipulated by a graphical 3-D CAD program.

Numerous further variations of the first aspect of the invention arepossible and may, for example, include combinations of the above notedvariations.

A second aspect of the invention provides a system for providingcomposite optical images to an eye or eyes of an observer wherein thecomposite optical images are included of first component optical imagesof an object along with second component optical images that are to bedisplayed in relationship to the first component optical images, thesystem including: (a) a beam splitter; (b) means for directing the firstcomponent optical images along a first optical path to a first beamsplitter and from the beam splitter along second and third optical pathsto a composite optical image viewing location and to an image capturedevice, respectively, wherein the image capture device provides firstelectronic data corresponding to the first component optical images; (c)means for using the first electronic data to provide second electronicdata corresponding to second component optical images to be viewed alongwith the first component optical images and providing said secondelectronic data to an electronic image display device to provide saidsecond component optical images; (d) means for directing the secondcomponent optical images along a fourth optical path to a second beamsplitter and then along a fifth optical path to the composite opticalimage viewing location such that the first component optical images andthe second component optical images are overlaid to provide saidcomposite optical images.

Numerous variations of the second aspect of the invention are possibleand include, for example: (1) the second beam splitter being the same asthe first beam splitter and the second and fifth optical paths being thesame optical path; (2) a sixth optical path extending from the secondbeam splitter to a second optical image capture device and whereincaptured second component optical image information provides at least aportion of the other data and is analyzed in combination with the firstelectronic data to produce said second electronic data; (3) the secondimage capture device being the same as the first image capture device;(4) the first component optical images includes optical images selectedfrom the group consisting of (a) physical optical images, (b) electronicoptical images, (c) emission optical images, and (d) hybrid opticalimages; (5) the second beam splitter is different from the first beamsplitter; (6) a third beam splitter that is different from both thefirst and second beam splitters and which is located along the fourthoptical path and which splits the fourth optical path into one branchthat continues to the second beam splitter and one branch that directsthe second component optical image onto a second image capture devicewhich in turn provides at least portion of the other data and isanalyzed in combination with the first electronic data to produce saidsecond electronic data; (7) the first component optical images includephysical optical images and wherein the system additionally includes alight source for directing light into an object positioning region thatis located at the beginning of the first optical path; (8) the firstcomponent optical images include physical optical images and wherein thesystem additionally includes a light source for directing light into anobject positioning region that is located at the beginning of the firstoptical path wherein the light source provides light selected from thegroup consisting of (a) visible light, (b) UV light, and (c) IR light;(9) the first component optical images include physical optical imagesand wherein the system additionally includes a light source fordirecting light into an object positioning region that is located at thebeginning of the first optical path, and providing one or more opticalcomponents between the light source and the object positioning regionwherein at least one of the one or more optical components is selectedfrom the group consisting of: (a) a lens, (b) a mirror, (c) a mechanicalor electronic shutter, (d) a prism, (e) a diffraction grating, (f) afilter; (10) providing one or more optical components located along oneor more of the first-fifth optical paths wherein the one or more opticalcomponents includes at least one component selected from the groupconsisting of: (a) a lens, (b) a mirror, (c) a shutter, (d) a prism, (e)a diffraction grating, and (f) a filter; (11) images reaching the beamsplitter via the first and fourth optical paths being provided in timemodulated forms such that images reaching the image capture device alongthe third optical path at selected times are the images carried alongthe first optical path but not the fourth optical path while at othertimes the images reaching the image capture device are the imagescarried along the fourth optical path but not the first optical path;(12) the images represented by the data provided along the secondelectronic path and displayed by the electronic image display device donot include images provided along the first optical path; (13) the usingof the first electronic data includes determining positioninginformation that places the second component optical images and thefirst component optical images within a desired tolerance by calculatinga display position for the second component optical images based, leastin part, on captured positions of the first component optical images,and wherein the calculation optionally includes spatial comparisons ofprior captured first component optical images and prior captured secondcomponent optical images; (14) means for adjusting an optical path byadjusting at least one component of the of system, wherein the componentis selected from the group consisting of (a) at least one beam splitter,(b) the display device, (c) the image capture device, (d) a positioningof the object, (e) an image focusing component located along one of theoptical paths, (f) a filter located along one of the optical paths, (g)a shutter located along one of the optical paths, (h) an aperture alongone of the optical paths, and (i) a mirror located along one of theoptical paths; (15) means for adjusting an optical path by adjusting atleast one component of the of system wherein the means for adjustingprovides for adjusting in a manner and at a time selected from a groupconsisting of: (a) manually during operation, (b) manually duringcalibration, (c) partially automatically during operation, (d) partiallyautomatically during calibration, (e) automatically during operation,and (f) automatically during calibration; (16) means for adjusting anoptical path by adjusting at least one component of the of systemwherein the means for adjusting provides for automatic adjustment via,at least in part, feedback provided by second component optical imagescaptured by the second image capture device; (17) the means for usingincludes processing the first electronic data using a programmedcomputer that is also programmed to control one or more systemcomponents and wherein at least one of the one more system componentsare selected from: (a) an electronic shutter located along one of theoptical paths, (b) modulation of a light source that irradiates theobject, (c) the image capture device, (d) the display device, (e) afocusing device located along one of the optical paths, (f) at least onestage coupled to at least one component for controlling one or more of aposition and orientation of the component, (g) an electronicallycontrolled aperture located along one of the optical paths, and (h) anelectronically controlled filter located along at least one of theoptical paths; (18) the composite optical images are provided during aprocedure selected from the group consisting of (a) a medical procedure,(b) a medical ophthalmic procedure, (c) an ophthalmic intraocular lensreplacement procedure, (d) an ophthalmic phacoemulsification procedure,(e) a medical restorative procedure, (f) a medical therapeuticprocedure, (g) a medical diagnostic procedure, (h) a medicalpreventative procedure, (i) a medical research procedure, (j) amanufacturing inspection procedure, (k) a manufacturing assemblyprocedure, and (l) a research or engineering visualization ordemonstration procedure; (19) the source or object includes an objectselected from the group consisting of (a) human tissue, (b) a humanorgan, (c) a human eye, (d) a rigid object, (e) a deformable object, (f)an expandable or contractible object and (g) an animate object, (h) aninanimate object; (19) the second component optical images include oneor more images selected from the group consisting of (a) capturedphysical object images, (b) magnetic resonance images, (c) coherencetomography image(s), (d) optical coherence tomography images, (e) acomputer generated polygonal graphic representation of an object, (f) animage that is in a format that can be manipulated by a graphicalanimation program, (g) an image that is in a format the can bemanipulated by a graphical 3-D CAD program.

Numerous further variations of the second aspect of the invention arepossible and may, for example, include combinations of the above notedvariations.

A third aspect of the invention provides a method of providing compositeoptical images to an eye or eyes (91) of an observer wherein thecomposite optical images are included of first component optical imagesof a source or object (81) along with second component optical imagesthat are to be displayed in relationship to the first component opticalimages, the method including: (a) providing the source or object that isto provide the first component optical image; (b) providing a beamsplitter (111); (b) providing an electronic image display device (141);(c) providing an image capture device (121); (d) providing a dataprocessing device (131); (e) directing light from the source of objectalong a first optical path (101) extending from an object to the beamsplitter and then from the beam splitter along a second optical path(102) to the eye or eyes of an observer and along a third optical path(103) to the image capture device, wherein the image capture devicecreates first electronic image data; (f) using, at least in part, thefirst electronic image data and additional information to produce secondelectronic image data corresponding to a second component optical imageand supplying said second electronic image data to the electronic imagedisplay device; (g) directing light from the electronic image displaydevice along a fourth optical path (104) to the beam splitter and fromthe beam splitter along the second and third optical paths to the eye oreyes of the observer and to the image capture device such that acomposite optical image is presented to the eye or eyes of the observer.

A fourth aspect of the invention provides a system for providingcomposite optical images to an eye or eyes (91) of an observer whereinthe composite optical images are included of first component opticalimages of an a source or object (81) along with second component opticalimages that are to be displayed in relationship to the first componentoptical images, the system including: (a) a beam splitter (111); (b) anelectronic image display device (141); (c) an image capture device(121); (d) a data processing device (131); (e) a first optical path(101) extending from a source or object positioning location to the beamsplitter; (f) a second optical path (102) extending from the beamsplitter to a composite optical image viewing location; (g) a thirdoptical path (103) extending from the beam splitter to an input of theimage capture device; (h) a first electronic signal path (151)connecting the output of the image capture device to a data processingdevice; (i) a second electronic signal path (152) connecting the outputof the data processing computer to an input of the electronic imagedisplay device; and (j) a fourth optical path (104) extending from anoutput of the electronic image display device to the beam splitter,wherein the first optical path transmits an optical image from thesource or object positioning location to the beam splitter and from thebeam splitter along both the second and third optical paths, wherein,based on one or more parameters associated with an image received by theimage capture device, and transmitted along the first electronic path,the data processing device provides data along the second electronicpath wherein the data represents images to be displayed by theelectronic image display device, wherein the fourth optical pathtransmits the second component optical images from the electronic imagedisplay device to the beam splitter where after the images are split andtravel along the second and third optical paths, and wherein the secondcomponent optical images and the first component optical images areoverlaid along both the second and third optical paths such that thesecond optical path provides the composite optical image to the viewinglocation while the third optical path provides both the second componentoptical image and first component optical image to the image capturedevice.

A fifth aspect of the invention provides a method of providing compositeoptical images to an eye or eyes (91) of an observer wherein thecomposite optical images are included of first component optical imagesof a source or object (81) along with second component optical imagesthat are to be displayed in relationship to the first component opticalimages, the method including: (a) providing the object (81) that is toprovide the first component optical images; (b) providing a first beamsplitter (211-1) and a second beam splitter (211-2); (c) providing anelectronic image display device (241); (d) providing an image capturedevice (221); (e) providing a data processing device (231); (f)directing light from the object along a first optical path (201)extending from a source or object to the first beam splitter and thenfrom the first beam splitter along both a second optical path (202) anda third optical path (203), wherein the third optical path leads to theimage capture device which creates first image data while the secondoptical path leads to the second beam splitter (211-2) and from therealong a fifth optical path (205) to the eye or eyes of an observer(203); (g) using, at least in part, the first image data and additionalinformation to produce second image data corresponding to secondcomponent optical images and supplying said second image data to theelectronic image display device; (h) directing light from the electronicimage display device along a fourth optical path (204) to the beamsplitter and from the beam splitter along the fifth optical path (205)to the eye or eyes of the observer.

A sixth aspect of the invention provides a system for providingcomposite optical images to an eye or eyes (91) of an observer whereinthe composite optical images are included of first component opticalimages of a source or object (81) along with second component opticalimages that are to be displayed in relationship to the first componentoptical images, the system including: (a) a first beam splitter (211-1);(b) a second beam splitter (211-2); (c) an electronic image displaydevice (241); (d) an image capture device (221); (e) a data processingdevice (231); (f) a first optical path (201) extending from an objectposition location to a the first beam splitter; (g) a second opticalpath (202) extending from the first beam splitter to the second beamsplitter and a third optical path (203) extending from the first beamsplitter to the image capture device; (h) a first electronic data path(251-1) extending from the image capture device to the data processingdevice and a second electronic data path (251-2) extending from the dataprocessing device to an electronic image display device; (i) a fourthoptical path extending from the electronic image display device to thesecond beam splitter; (j) a fifth optical path extending from the secondbeam splitter to a composite optical image viewing location wherein thefifth optical path carries images from the second and forth opticalpaths to said composite optical image viewing location, wherein the dataprocessing device provides second electronic image data for display bysaid display device wherein the second electronic image data is derivedat least in part using data generated by the image capture device.

A seventh aspect of the invention provides a method of providingcomposite optical images to an eye or eyes (91) of an observer whereinthe composite optical images are included of first component opticalimages of a source or object (81) along with second component opticalimages that are to be displayed in relationship to the first componentoptical images, the method including: (a) providing the source or object(81) that is to provide the first component optical images; (b)providing a first beam splitter (311-1) and a second beam splitter(311-2); (c) providing an electronic image display device (341); (d)providing an image capture device (321); (e) providing a data processingdevice (331); (f) directing light from the source or object along afirst optical path (301) extending from the object to the first beamsplitter and then from the first beam splitter along a second opticalpath (302) to the second beam splitter (211-2) and then along both afifth optical path (305) to the eye or eyes of an observer and along athird optical path (303) to the image capture device, wherein the imagecapture device creates first image data; (g) using, at least in part,the first image data and additional information to produce second imagedata corresponding to second component optical images and supplying saidsecond image data to the electronic image display device; (h) directinglight, in the form of the second component optical image, from theelectronic image display device along a fourth optical path (204) to thefirst beam splitter and from the first beam splitter along the second,fifth, and third optical paths, wherein the fifth optical path providesa composite optical image of the second component optical images and thefirst component optical images to the eye or eyes of the observer andthe third optical path provides the second component optical images tothe image capture device and wherein data obtained from the imagecapture device from at least a portion of the second component opticalimages provides at least a portion of the additional information.

An eight aspect of the invention provides a system for providingcomposite optical images to an eye or eyes (91) of an observer whereinthe composite optical images are included of first component opticalimages of a source or object (81) along with second component opticalimages that are to be displayed in relationship to the first componentoptical images, the system including: (a) a first beam splitter (311-1);(b) a second beam splitter (311-2); (c) an electronic image displaydevice (341); (d) an image capture device (321); (e) a data processingdevice (331); (f) a first optical path (301) extending from an objectposition location to the first beam splitter; (g) a second optical path(302) extending from the first beam splitter to the second beamsplitter; (h) a fifth optical path (305) and a third optical path (303)extending from the second beam splitter to a viewing location and to animage capture device respectively; (i) a first electronic data path(351-1) extending from the image capture device to the data processingdevice and a second electronic data path (251-2) extending from the dataprocessing device to an electronic image display device; (j) a fourthoptical path extending from the electronic image display device to thefirst beam splitter; wherein the second, fifth, and third optical pathscarry both the first component optical images and the second componentoptical images such that composite optical images are presented at theviewing location; and wherein the data processing device provides secondelectronic image data for display by said electronic image displaydevice wherein the second electronic data is derived at least in partusing data generated by the image capture device.

A ninth aspect of the invention provides a method of providing compositeoptical images to an eye or eyes (91) of an observer wherein thecomposite optical images are included of first component optical imagesof a source or object (81) along with second component optical imagesthat are to be displayed in relationship to the first component opticalimages, the method including: (a) providing a source or object (81) thatis to provide first component optical images; (b) providing a first beamsplitter (411-1), a second beam splitter (411-2), and a third beamsplitter (411-3); (c) providing an electronic image display device(441); (d) providing a first image capture device (421-1) and a secondimage capture device (421-2); (e) providing a data processing device(431); (f) directing light from the source or object along a firstoptical path (401) extending from the object to the first beam splitterand then along a second optical path (402) and a third optical path(403) wherein the second optical path extends from the first beamsplitter to the second beam splitter while the third optical pathextends from the first beam splitter to the first image capture device;(g) directing light reaching the second beam splitter from the secondoptical path along a fifth optical path (405); (h) converting imagesreceived by the first image capture device into first image data andusing, at least in part, the first image data and additional informationto produce second image data corresponding to second component opticalimages and supplying said second image data to the electronic imagedisplay device; (i) directing light from the electronic image displaydevice along a fourth optical path (404) to the third beam splitter andfrom there along both a sixth and a seventh optical path, wherein thesixth optical path leads to the second beam splitter and then to thefifth optical path, wherein the second component optical images and thefirst component optical images form composite optical images that arepresented to the eye or eyes of an observer, while the seventh opticalpath extends from the third beam splitter to a second image capturedevice which provides electronic image data to the data processingdevice which is at least in part used as a portion of the additionalinformation.

A tenth aspect of the invention provides a system for providingcomposite optical images to an eye or the eyes (91) of an observerwherein the composite optical images are included of first componentoptical images of a source or object (81) along with second componentoptical images that are to be displayed in relationship to the firstcomponent optical images, the system including: (a) a first beamsplitter (411-1); (b) a second beam splitter (411-2); (c) a third beamsplitter (411-3); (d) an electronic image display device (441); (e) anfirst image capture device (421-1); (f) a second image capture device(421-2); (g) a data processing device (431); (h) a first optical path(201) extending from an object position location to the first beamsplitter; (i) a second optical path (202) extending from the first beamsplitter to the second beam splitter and a third optical path (203)extending from the first beam splitter to the first image capturedevice; (j) a first electronic data path (251-1) extending from theimage capture device to the data processing device and a secondelectronic data path (251-2) extending from the data processing deviceto an electronic image display device; (k) a fourth optical pathextending from the electronic image display device to the third beamsplitter; (l) a sixth and seventh optical path extending from the thirdbeam splitter to the second beam splitter and the second image capturedevice, respectively; (m) a fifth optical path extending from the secondbeam splitter to a composite optical image viewing location wherein thefifth optical path carries images from the second and forth opticalpaths to said composite optical image viewing location, wherein the dataprocessing device provides second electronic image data for display bysaid display device wherein the second electronic data is derived atleast in part using data generated by the first and second image capturedevices.

Numerous variations to the third-tenth aspects of the invention arepossible and include for example those noted above with regard to thefirst and second aspects of the invention.

An eleventh aspect of the invention provides a method of providingcomposite optical images to an eye or eyes of an observer wherein thecomposite optical images are included of first component optical imagesof a source or object along with second component optical images thatare to be displayed in relationship to the first component opticalimages, the method including: (a) directing first component opticalimages from a source or object to a first beam splitter and thereafterdirecting split first component optical images to a first image capturedevice located along a first image capture path and to a viewinglocation located along a first viewing location path; (b) directingsecond component optical images from an electronic image display deviceto a second beam splitter and thereafter directing split secondcomponent optical images to a viewing location along a second viewinglocation path having at least a terminal portion that overlays aterminal portion of the first viewing location path; (c) using at aportion of data created from the first component optical image capturedby the first image capture device in generating electronic image datathat is provided to an electronic image display device to create updatedsecond component optical images.

Numerous variations of the eleventh aspect of the invention are possibleand include, for example: (1) the second beam splitter being the same asthe first beam splitter; (2) the first component optical images includeoptical images selected from the group consisting of (a) physicaloptical images, (b) electronic optical images, (c) emission opticalimages, and (d) hybrid optical images; (3) the second beam splitterbeing different from the first beam splitter; (4) the first componentoptical images include physical optical images and the methodadditionally includes directing light from a light source onto an objectlocated at the beginning of the first optical path that gives rise tothe physical optical images and wherein the light source optionallyproduces light selected from the group consisting of (a) visible light,(b) UV light, and (c) IR light; and wherein the method optionallyincludes providing one or more optical components between the lightsource and the object wherein at least one of the one or more opticalcomponents is selected from the group consisting of: (a) a lens, (b) amirror, (c) a mechanical or electronic shutter, (d) a prism, (e) adiffraction grating, and (f) a filter; and wherein the method optionallyproviding one or more optical components located along a path traveledby either one or both of the first component optical image and thesecond component optical image, wherein the one or more opticalcomponents includes at least one component selected from the groupconsisting of: (a) a lens, (b) a mirror, (c) a shutter, (d) a prism, (e)a diffraction grating, and (f) a filter; (5) at least one of the firstcomponent optical image or second component optical image reaching thebeam splitter are provided in time modulated forms such that imagesreaching an image capture device at a given times are only firstcomponent optical images while at other times are the images reaching animage capture device are only second component optical images; (6) thesecond component optical images do not include image that are the sameas first component optical images.

A twelfth aspect of the invention provides a system for providingcomposite optical images to an eye or eyes of an observer wherein thecomposite optical images are included of first component optical imagesof a source or object along with second component optical images thatare to be displayed in relationship to the first component opticalimages, the method including: (a) means for directing first componentoptical images from a source or object to a first beam splitter andthereafter directing split first component optical images to a firstimage capture device located along a first image capture path and to aviewing location located along a first viewing location path; (b) meansfor directing second component optical images from an electronic imagedisplay device to a second beam splitter and thereafter directing splitsecond component optical images to a viewing location along a secondviewing location path having at least a terminal portion that overlays aterminal portion of the first viewing location path; (c) means for usingat a portion of data created from the first component optical imagecaptured by the first image capture device in generating electronicimage data that is provided to an electronic image display device tocreate updated second component optical images.

Numerous variations to the twelfth aspect of the invention are possibleand include for example those noted above with regard to the eleventhaspect of the invention and those note above with regard to the secondaspect of the invention.

Other aspects of the invention will be understood by those of skill inthe art upon review of the teachings herein. Other aspects of theinvention may involve combinations of the above noted aspects of theinvention. These other aspects of the invention may provide variouscombinations of the aspects presented above as well as provide otherconfigurations, structures, functional relationships, and processes thathave not been specifically set forth above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a block diagram showing sample components and samplecomponent relationships according to a system of a first embodiment ofthe invention wherein the system provides for optical formation of acomposite optical image using a single beam splitter to direct apreliminary optical image (e.g. a physical optical image, a firstelectronic optical image, an emission optical emission, or hybrid image)and an electronic optical image (e.g. a second electronic optical image)along both a viewing optical path and along an image capture opticalpath.

FIG. 2 provides a block diagram showing sample components and samplecomponent relationships according to a system of a second embodiment ofthe invention wherein the system provides for optical formation of acomposite optical image using two beam splitters wherein the first beamsplitter directs a branch of a first component optical image (e.g. aphysical optical image, a first electronic optical image, an emissionoptical image, or a hybrid optical image) along an image capture pathand the second beam splitter directs both a branch of the firstcomponent optical image and a branch of a second component optical image(e.g. second electronic optical image) along a viewing optical path.

FIG. 3 provides a block diagram showing sample components and samplecomponent relationships according to a system of a third embodiment ofthe invention wherein the system provides for optical formation of acomposite optical image using two beam splitters wherein the first beamsplitter merges components of a first component optical image (e.g. aphysical optical image, a first electronic optical image, an emissionoptical image, or a hybrid optical image) and a second component opticalimage (e.g. a second electronic optical image) along a single opticalpath that proceeds to the second beam splitter which divides the mergedoptical images between an image capture path and a composite opticalimage viewing path.

FIG. 4 provides a block diagram showing sample components and samplecomponent relationships according to a system of a fourth embodiment ofthe invention wherein the system provides for optical formation of acomposite optical image using three beam splitters wherein the firstbeam splitter directs a branch of a first optical image (e.g. a physicaloptical image, a first electronic optical image, an emission opticalimage, or a hybrid optical image) along a first image capture path andthe second beam splitter directs both a branch of the split firstoptical image and a branch of a split second component optical image(e.g. a second electronic optical image) as a composite optical imagealong a viewing optical path and wherein a third beam splitter directsseparate branches of the second component optical image (e.g. secondelectronic optical image) along a second image capture path and to thesecond beam splitter.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Definitions

As used herein an “optical image” is a visual reproduction of an objectusing an optical system. The optical system may involve light reflectedfrom or transmitted from a source directly to the eye of an observer orit may involve such reflected or transmitted light as modified by anexternal system of mirrors, lenses, apertures, shutters, and the like(e.g. from reflected, refracted, and/or diffracted light waves). Theoptical image may originate from a diffuse source of radiation (e.g. aphysical object having a microscopically rough surface from whichradiation is reflected) or from an electronically controlled radiatingor emission source (e.g. an LED, backlit LCD, CRT, or other imageproducing emission source which is driven by image data that has beenelectronically transmitted to the display). An optical image produceddirectly by a diffuse source without an intermediate emission sourcealong the optical path shall be termed a “physical optical image” whilean optical image produced by an electronic emission source without anintervening diffuse reflector along the optical path shall be termed an“electronic optical image”. An optical image produced by an emissionsource that is a non-electronic emission source, shall be termed an“emission optical image” An optical image produced by at least two of adiffuse reflective source, and electronic emission source, and anon-electronic emission source, shall be termed a “hybrid opticalimage”.

As used herein an “optical path” is a path along which light wavestravel from one position to another and along which the light waves mayencounter optical elements that shape or define an image (e.g. size,focal plane, contrast, wavelengths, and the like) or the actual path oftravel itself and wherein the optical elements may include, e.g. one ormore of lens, mirrors, diffraction gratings, prisms, apertures, filters,shutters, and the like)

As used herein an “electronic image” is an image produced by translationof electronic data into a visual representation wherein the data iscreated, stored or transmitted in in a form requiring electronicinterpretation (e.g. a series of binary voltage, current, magnetic, oreven optical pulses) and is transformed into a visual image forperception by the eye using some form of electrical or electroniccontrolled display device. An electronic image may be a captured imageof a physical object (i.e. a diffuse reflector), a captured image from anon-electronic emission source, a captured image from an electronicsource, or a captured image from a hybrid emission, or an image createdby a computer (e.g. produced by CAD software, computer animationsoftware), or a combination of these.

As used herein a “component optical image” is one of a physical opticalimage, an emission optical image, an electronic optical image, or ahybrid optical image.

As used herein “composite optical image”, “overlaid optical image”, and“merged optical image” unless specifically indicated otherwise aresubstantially synonymous terms that refer to optical images that arecreated by the spatial optical merging of at least two component opticalimages, i.e. first and second optical images wherein the first opticalimages includes one of a physical optical image, a first electronicoptical image, an emission optical image, or a hybrid optical image andwherein the second optical image is a an electronic optical image (e.g.a second electronic optical image). Temporal merging or simultaneity ofcomponent optical images of a composite optical image (e.g. the displayof both physical optical and electronic optical images simultaneously)may or may not exist. In some embodiments, simultaneous and evencontinuous display of both the first and second optical images may existwhile in other embodiments, the first and second images willalternatively be turned on and off along their respective optical pathsto provide a perceived temporal merging of the images to the eye(s) of aviewer but temporally separated images to an image capture device.

As used herein a “electronic display device” or an “electronic imagedisplay device” is any device that converts electronic data (e.g.digital data) into an optical image that may be viewed directly orindirectly by the eye of an observer or that may be transmitted along adesired optical path. The display device may be of a digital form (e.g.LED display, LCD display) or analog form (e.g. CRT display).

As used herein an “image capture device” is a device that captures thespatial configuration of an image and translates that the image intodata that may include not only spatial information but color, intensity,and temporal information. The captured image may be supplied to adisplay device directly or not at all and/or it may be supplied to adata processing device such as a digital signal processor or aprogrammed computer for analysis or manipulation. An image capturedevice may take a variety of forms such as, for example, a digitalcamera, a digital video camera, a CCD array, a photocell over whichlight is scanned or which is translated to sequentially read differentportions of an image. Such devices may be configured to capturemacro-scale images or microscopic images.

As used herein “an electronic path” is a functional connection overwhich electronic signals are passed from one electronic device toanother. The electronic path may take the form of a current carryingwire or electromagnetic signal that begins at one device and is receivedby a second device at the end of the electronic path. The electronicpath may include numerous electronic components that shape, strengthen,or other manipulate an electronic signal as it move from the firstelectronic device to the second electronic device.

As used herein “beam splitter” is intended to be a generic term coveringvarious optical elements and optical element combinations that can beused to (1) split a beam or an optical image so that the split beam oroptical image travels along at least two different paths and (2) combinethe two appropriately presented beams or images such that the combinedbeams or images (i.e. composite optical images) travel along one or morecommon paths. A beam splitter, for example, may be formed from apartially silvered mirror that is located appropriately relative to anincoming optical path (e.g. at 45 degrees to the incoming path), bymultiple partially silvered mirrors, by a cube or other geometric shapeformed from two or more triangular prisms. Beam splitter technology iswell known in the art and is described in, for example, U.S. Pat. No.7,701,583.

Embodiments in General

The present disclosure sets forth four primary, specific, and exemplarysystem embodiments of the invention. Each of these primary embodimentsinvolves the viewing of a composite optical image formed by at least twodistinct component optical images (a first component optical image thatis one of a physical optical image, an electronic optical image, anemission optical image or a hybrid optical image) and a second componentoptical image that is an electronic optical image that is presented inrelationship to the first component optical image to provide an enhancedor augmented visual representation associated with the first componentoptical image. In some of these embodiments, the first component opticalimage is a physical optical image while the second component optical isan electronic optical image. In other embodiments the first componentoptical image is an electronic optical image while the second componentoptical image is also an electronic optical image. The first embodimentrequires at least one beam splitter, the second and third embodimentsrequire at least two beam splitters, while the fourth requires at leastthree beam splitters. Each embodiment also requires at least one or twoimage capture devices, at least one data processing device, and at leastone electronic display device. Variations of each embodiment may add inadditional electronic components, optical components, and mechanicalcomponents as well as a variety of different hardwired or programmedcontrol and monitoring elements to allow real time feedback andenhancement of system performance and/or information logging.

As will be discussed in further detail below, embodiments of the presentinvention are based, in part, upon the use of an optical pathsplitting/merging arrangement (i.e. using at least one beam splitter) toprovide splitting of individual component optical images (e.g. intoviewing and capture paths) while providing a viewing path along which acomposite optical image is presented and possibly providing a compositeoptical image path for image capture purposes as well. The systems andmethods of the various embodiments of the invention also include, inaddition to at least one beam splitter, at least one image capturedevice, at least one electronic image display device, and at least onedata processing device. Embodiments, as appropriate, may also includelighting sources for illuminating physical objects to be viewed andvarious optical elements for guiding and controlling images as theypropagate through the system. These lighting sources may provide, forexample, visible, IR, or UV lighting and the resulting images may bephysical optical images, emission optical images, or hybrid opticalimages. Captured images may be used to provide alignment or overlayfeedback that can be used in providing improved image merging and thusimproved augmentation.

The issued US patents, patent applications, published foreignapplications, and other published references that are cited herein arehereby incorporated by reference in their entirety to the same extent asif each was specifically and individually indicated to be incorporatedby reference.

The various exemplary systems and methods set forth herein utilizeaugmented reality algorithms to generate electronic optical images andthus composite optical images. Some embodiments of the invention providereal-time electronic optical image feedback relative to physical opticalimages this feedback can be used to provide real-time calibration of thesystem and improved presentation of composite optical images. Thisfeedback may be used to improve subsequent image presentation in areactive manner and/or in a predictive manner.

Image capture devices, or “image capture means”, appropriate for thevarious embodiments of the presentation invention can take on a varietyof forms. Examples of such image capture devices include photocellarrays, cameras (e.g. digital cameras or camcorders), charged-coupleddevices (CCD), and CMOS Cameras. In such embodiments, the frame rate ofthe image capture device and of an electronic image display device maybe synchronized to allow for the imaging device to view either only asingle component optical image, subset of component optical images, orall component optical images as a composite optical image.

As noted above, in some embodiments of the invention the image capturedevice may capture images at a particular frame and the electronic imagedisplay device may display images at a particular frame rate. Forexample the image capture device may detect images at a frame rate of30-1000 frames per second (FPS) and in particular embodiments the framerate may be selected to be between desired narrow band, e.g. between30-100 FPS, between 100-200 FPS, and the like.

Electronic image display devices, or electronic image display means,appropriate for the various embodiments of the invention can take on avariety of forms. Examples of such devices, or means, include backlitliquid crystal (LCD) displays, plasma displays, cathode ray tubes(CRTs), digital light projectors (DLPs), light emitting diodes (LEDs),and Transparent Organic Light Emitting Diodes (OLEDs).

In different embodiments, the image capture device may view a firstcomponent optical image (e.g. physical optical image) alone, a secondcomponent optical image (e.g. an electronic optical image alone), aplurality of component optical images but less than all optical images(e.g. two out of three, three out of four, two out of four, and thelike) or a composite optical image. Capturing of a first componentoptical image (e.g. physical optical image) alone may occur by activelytransmitting the first component optical image along its optical imagepath while blocking a second component optical image, arranged to travelon a different optical path from the first component optical image, fromreaching a beam splitter that would result in merging and thentransmission to the image capture device. This may be done, for example,by synchronizing the triggering of the frame capture rate of the imagecapture device and frame emission from the electronic imaging devicewith an appropriate phase difference. Alternatively this may beaccomplished by shuttering the second component optical image fromreaching the beam splitter while the image capture device is to capturethe first component optical image (e.g. physical optical image) (e.g.using an acousto-optic modulator or other fast shuttering device).Capturing the second optical image (e.g. an electronic optical image orsecond electronic optical image) alone may be achieved by shuttering thefirst component path (e.g. the physical optical image path) or byturning off or shuttering illuminating light from reaching a source ofthe first image while the image capture device is capturing the secondcomponent optical image. If the electronic image display device providesshort display pulses, it may be necessary to trigger the display deviceemission with acquisition triggering of the image capture device (e.g.have them operate in phase with one another). When images from both thepaths are allowed, or made, to exist simultaneously, direct capture ofcomposite optical images is possible.

Various embodiments of the invention may make use of optical componentsat a composite optical image viewing location to allow one or moreobservers or operators to view the composite optical image. Examples ofsuch optical components include microscopes, ocular indirect virectomylens, ocular Landers wide angle surgical viewing system. In someembodiment variations, the composite optical image may be projected onto a display screen as opposed to directly entering the observer's eyeor eyes.

In some embodiments or embodiment variations as will be discussedhereafter, the composite optical image resulting from the overlaying ofa first component optical image (e.g. physical object image) and asecond component optical image (i.e. an electronic optical image) may ormay not be initially spatially correlated but will become so quicklywith successive image presentations as differences in position ofoverlaid component optical images are ascertained and used to providefeedback control to the configuration of, positioning of and/ororientation of system components and/or display position of the secondcomponent optical image on the electronic image display device.

In some embodiments of the invention the data processing device(s) use atechnique called markerless tracking to track the second componentoptical image (e.g. a first or second electronic optical image) relativeto the first component optical image (e.g. physical optical image or afirst electronic component optical image) and by comparing the positionsof the second component optical image relative to known desiredpositions of second component optical image features relative to actualfirst component optical image features, one or more corrections can beidentified and implemented and thereafter additional images captured,additional processing done, and additional corrections made as necessaryto bring and keep the relative positions of the component optical imagesin appropriate positions. These processes may or may not incorporatepredictive techniques (based on past errors in positioning) and/oriterative techniques to yield better and faster merging of the imageswithin desired tolerances.

In some embodiments, involving certain first component optical images(e.g. physical optical images) and certain second component opticalimages (i.e. electronic optical images), it may be possible that thesecond component optical images are not a good candidate for markerlesstracking. In such instances, fiduciary marks may be introduced into thesecond component optical image to overcome this limitation. As usedherein, “fiduciary marks” are objects inserted into an image and used asa point of reference. The fiduciary marker is either placed into or onan image. Fiduciary marks could be parts of the actual overlay or added.Those marks could be tracked using algorithms such as Scale InvariantFeature Transform (SIFT) or Kanade-Lucas-Tomasi (KLT) feature trackers.Another method would be to directly compare known featurecorrespondences between the first component optical images and thesecond component optical images to compute misalignment amounts orcorrection factors.

In addition to the possibility of ensuring alignment and orientation ofthe first and second component optical images (and any additionalcomponent optical images) via feedback, another job for the dataprocessing device or devices (e.g. computer or computers) may be tomanipulate existing electronic optical images (e.g. position, orient,scale, color data associated with such images) or to create andmanipulate electronic optical images using augmented reality algorithmswhich are well known in the field. Examples of such algorithms aredescribed in (1) Kato, H., Billinghurst, M. “Marker tracking and hmdcalibration for a video-based augmented reality conferencing system”;(2) In Proceedings of the 2nd IEEE and ACM International Workshop onAugmented Reality (IWAR 99), October 1999; (3) R. Azuma, A Survey ofAugmented Reality Presence: Teleoperators and Virtual Environments, pp.355-385, August 1997; and (4) P. Milgram and A. F. Kishino, Taxonomy ofMixed Reality Visual Displays IEICE Transactions on Information andSystems, E77-D(12), pp. 1321-1329, 1994.

In some embodiments of the invention, a combination of markless tracking(to ascertain first and second component optical image positions and/ororientations) and augmented reality algorithms are used to yield datafor a next electronic optical image to be displayed. Markerless trackingalgorithms are known in the art and are described for example, in (1) L.Vacchetti, V. et al. IEEE Transactions on Pattern Analysis and MachineIntelligence, 26 (10): 1385-1391, 2004; (2) L. Vacchetti et al.“Combining Edge and Texture Information for Real-Time Accurate 3D CameraTracking,” International Symposium on Mixed and Augmented Reality,Arlington, Va., November 2004; (3) L. Vacchetti, V. et al. “Stable 3—DTracking in Real-Time Using Integrated Context Information,” Conferenceon Computer Vision and Pattern Recognition, Madison, Wis., June 2003;(4) V. Lepetit et al. “Fully Automated and Stable Registration forAugmented Reality Applications,” International Symposium on Mixed andAugmented Reality, Tokyo, Japan, 2003.

Some embodiments of the invention and variations thereof, use programmedalgorithms or logic for aligning component optical images and to render(i.e. convert from graphic data to visual form) or even createelectronic optical images for presentation to the electronic imagedisplay device while other embodiment may limit such programmedalgorithms to only the rendering and creation activities.

In some embodiments, an ideal augmented reality image (e.g. an imagethat consists of a physical optical image component and an electronicoptical image component) may be created within the computer and it maybe used for comparison to captured composite optical images to determineif a composite optical image has been properly created and/or presented.In these embodiments, it may be possible to determine an alignment errorin the components of the composite optical image. Correction informationcan then be calculated and used to adjust the position of the electronicoptical image output by electronic image display device, e.g. by use ofa Euclidean Transformation. These and other techniques are described inYi et al. An Invitation to 3-D Vision; Hartley, R.˜I. and Zisserman, A.“Multiple View Geometry in Computer Vision”, Cambridge tions may beperformed by the separate manipulation device. University Press, 2000.In some embodiments, data manipulations may be performed by a datamanipulation device that is incorporated into the image capture devicewhile in other embodiments, a portion of the data manipulations mayperformed by a separate data manipulation device, while in otherembodiments, all data manipula.

In some embodiments, misaligned first component optical images (e.g.physical optical images) and second component optical images (i.e.electronic optical images) may be corrected by the physical movement ofor reconfiguration of system components alone, by display/imagepositioning adjustments, or by a combination. The physical movement ofsystem components may involve, for example, movement of the displaydevice, the image capture device an/or the beam splitter whilereconfigurations may involve, for example, opening and closing ofapertures (e.g. irises) and setting adjustments for focusing components.

In some embodiments, a single computer source or distributed computingmay be used to derive necessary data and/or to control movements and/oroperation of system components. Computing capability may exist in thedisplay device and/or in the image capture device.

The correcting of unaligned first component optical images (e.g.physical optical images) and second component optical images (i.e.electronic optical images) by adjustment of the display position onelectronic image display device may occur by aligning the secondcomponent optical image to the first component optical image usingre-rendering of the electronic optical image based on a mathematictransformation as represented by the following equation:

Y=f(U)

where U is a vector characterizing the size, location, and orientationof an electronic optical image as shown on the display device. Y is avector characterizing the size, location, and orientation of theelectronic optical image as seen by the image capture device. Themathematical model “f” maps U to Y, where f is a mathematicaltransformation. Such transformations include linear, affine, homography,nonlinear (strictly speaking homography is nonlinear), look-up tables,and the like.

In augmented reality applications, desired second component opticalimages, denoted in the following equation as R, are generated to besuperimposed with first component optical images, denoted as T, withdesired geometrical alignment. The rendering of the electronic opticalimage R requires the inverse model describing the transformation betweenthe viewing device and the display device:

U=f′(R)

When the mathematical model of the second component optical image andits inverse do not accurately represent the actual physical system,error occurs such that the second optical image and the first opticalimage (e.g. electronic optical image and the physical optical image) arenot aligned as desired. To ensure accurate alignment between the firstcomponent optical images and the second component optical images as seenby the viewing device, the feedback mechanism may capture the renderedcomposite optical images. The mechanism may then compare the idealcomposite optical image and the captured composite optical image andcalculates the geometrical alignment error. The geometrical alignmenterror is the vector (3-D displacement and orientation and image size ascharacterized by appropriate camera image projection model) between theideal composite optical image and the measured composite optical image.This error can be mathematically represented as follows:

E(k)=R−Y(k),

where R is the desired second component optical image rendering vector,which will align with the first component optical image T in a desirablemanner determined by the augmented reality software. Y(k) is themeasured electronic optical image with iteration index number k. Theerror vector E(k) is used to make correction for the composite opticalimage rendering in the following iterative feedback correction:

U(k+1)=U(k)+g(E(k))

where U(k) is the present rendering vector of the composite opticalimage, U(k+1) is the next feedback compensated rendering vector of thecomposite optical image. The function g is ideally the inverse functionf′. When the inverse model is not readily available, g can be a gainmatrix that is tuned to assure that the error E(k) diminishes toacceptable level over a number of iterations.

In some embodiments, the alignment of the composite optical image isadjusted by adjusting the rendering of the second component opticalimage onto the electronic image display device. As an example,adjustment of the rendering of the second component optical image can beaccomplished using the following methodology. First the viewpoint fromwhich a 3-D model of the second component optical image is to bedisplayed is computed. Then, from the computed viewpoint, the surface ofthe 3D model is projected onto the image plane of the electronic imagedisplay device using an appropriate virtual camera model (e.g. pinhole).It is also possible to add further rendering steps by distorting theprojected 3-D model using a mathematic transformation directly on thepixels. (linear, affine, nonlinear, homography).

The following is an illustrative example of feedback. Let U representthe position and orientation of a 3-D structure represented by thesecond component optical image in the coordinates of the electronicimage display device. The second component optical image is thenrendered on the electronic image display device using a projective modelas,

u=Π _(d) q _(d)(U)

where Π_(d) is the virtual display projection, q_(d)(U) is a EuclideanTransformation (rigid body motion) and u is the second component opticalimage on the display. The rendered overlay on the display is thencaptured by the image capture device by another projectivetransformation

Y=Π _(c) q _(c)(u)

where Π_(c) is the image capture device projection, q_(c)(u) is aEuclidean Transformation and y is the image of the second componentoptical image. Let R represent the desired location of the secondcomponent optical image in the image capture device coordinates suchthat it is overlaid correctly with the first component optical image. Ris computed using the location of the first component optical image incoordinates of the image capture device as obtained via markerlesstracking. The second component optical image can also be tracked and itsvirtual location in the camera coordinates can be computed as Y. Thefeedback update law, U(k+1)=U(k)+g(E(k)), can then be used to adjust Uto reduce E to an acceptable level.

First Specific Embodiment

The first specific embodiment provides an augmented reality system 100and method allowing an observer to view first component optical images(e.g. physical optical images), that are initially provided on a firstoptical path, that are combined with second component optical images(e.g. electronic optical images), that are originally provided on afourth optical path, wherein the overlaying of the two component opticalimages occurs optically via a beam splitter and wherein branches of thefirst component optical images and the second component optical imagesare sent by the beam splitter along a second optical path for viewing byan observer or operator and along a third optical path into an imagecapture device wherein the captured copies of the first componentoptical images are used, at least in part, to create or position thesecond component optical images to be displayed as part of the compositeoptical image. In some variations of this embodiment, feedback of therelative positions of the captured first component optical images andthe captured second component optical images are used to enhancealignment of the images in producing the composite optical image. Insome variations of the first embodiment, the creation or transmission ofthe first component optical images and the second component opticalimages along the first and fourth optical paths, respectively, aremodulated such that at selected times only one of the two componentimages reaches the image capture device such that capture of individualcomponent images and individual processing of those images can be usedto provide enhanced alignment when forming the composite optical images.In some variations of the first embodiment, enhanced alignment occursmanually during a set up procedure, manually during usage,semi-automatically during a set up procedure, semi-automatically duringusage, automatically during a set up procedure, or during usage.

The features of, and various alternatives to, the first embodiment canbe better understood by reference to FIG. 1 which provides a blockdiagram showing sample components and sample component relationshipsaccording to a system of a first embodiment of the invention wherein thesystem provides for optical formation of a composite optical image usinga single beam splitter to direct first component optical images andsecond component optical images along both a viewing optical path (pathand along an image capture optical path.

The system of FIG. 1 includes an source or object placement location 82where the source or object 81 is to be located with a first optical path101 extending from the source object located within the placementlocation 82 and a beam splitter 111. First component optical images(e.g. physical optical images) are transmitted along path 101 and arethen being split into first and second components which continue along asecond optical path 102 and third optical path 103 respectively. Thesecond optical path 102 extends from the beam splitter 111 to a viewinglocation 92 wherein the eye or eyes 91 of an observer may see the firstcomponent optical image of the object or source 81 (when such an objectis appropriately located or a source is appropriately activated) whilethe third optical path 103 extends from the beam splitter to the animage capture device 121 and carries first component optical images,similar to that carried along path 102, to the image capture device.Based on this configuration a first component optical image of an objector source 81 can be simultaneously seen at the end of the second opticalpath 102 or captured at the end of the third optical path 103. The firstcomponent optical image captured by the image capture device 121 can beconverted to electronic data and transmitted along electronic path 151to a data processing device (e.g. to a digital signal processor forhardwired manipulations or to a programmed computer for more readilyupdatable manipulations) for determination or characterization ofselected features (e.g. determination of object orientation, position,and size). This information can in turn be used in preparing data forrepresentation of second component optical images, from an electronicimage display device, that is to be displayed in relative position tothe first component optical images in a composite optical image that isto be created.

In some embodiment variations, it may be necessary to present secondcomponent optical imagers, e.g. electronic optical images, in reversedconfigurations (e.g. mirror image configurations) relative to firstcomponent optical configurations such that when composite optical imagesreach the eye or eyes of an observer or when they reach the imagecapture device they will have proper overlaid configurations. Suchreversals may be provided via the electronic data and driving of theelectronic display device or via optical components located along one ormore of the optical paths. Similarly, depending on what purpose may beassociated with the composite optical image viewing, it may be desirableto insert additional optical elements to ensure that the viewedcomposite optical image has a desired orientation relative to the firstcomponent optical images that are creating a portion of the compositeoptical image.

Electronic image data is sent along electronic path 152 to theelectronic image display device 141 (e.g. an back lit LCD array, an LEDarray, a laser or broad area light source whose output can becontrollably scanned, filtered, and modulated) whereafter the createdelectronic optical image (i.e. second component optical image) istransmitted along optical path 104 wherein the image encounters beamsplitter 111 at a different angle or face than that of optical path 101(e.g. the two paths are substantially perpendicular to one another).Upon reaching beam splitter 111 the second component optical image issplit into two components, as was the first component optical image,with one branch traveling along second optical path 102 to viewinglocation 92 and the other traveling along the third optical path 103 toimage capture device 121 whereby composite optical images are formedalong both optical paths 102 and 103. Of course, depending on thequality of alignment (e.g. size, position, orientation, focus, and eventiming) of the first and second component optical images traveling alongpaths 101 and 104, respectively, the resulting overlap of the compositeoptical images reaching the viewing location 92 and the image capturedevice 121 after traveling along paths 102 and 103 may be more or lesscorrelated.

In some variations of the first embodiment of the invention, one or moreof the electronic image display device 141, the beam splitter 111, theelectronic image capture device 121, and possibly the viewing location92 and/or the positioning location 82 may be repositionable orreorientable relative to the other components either manually undercomputer control to help provide improved alignment of composite opticalimage components (i.e. of the first and second component optical imagesthat make up the composite optical images). This repositioningcapability may be achieved by placing these components on positioningstages ranging from single to six axis stages. These stages may bemanually adjustable or computer controlled (e.g. for control by the dataprocessing computer or by some other controller incorporated into thesystem).

In some variations of the first embodiment, additional optical elementsmay be located along one or more of the first-fourth optical paths asindicated by blocks 171-1 to 171-4 in FIG. 1. These optical elements mayinclude apertures, lens, mirrors, filters, shuttering devices (e.g.acousto-optic modulators), additional beam splitters, and the like thatmay be used to position, size, orient, modulate, split, combine, or varylight wavelengths reaching the viewing or capture positions. Suchoptical elements may be configurable manually or by computer controlledadjustment.

In some embodiment variations, one or more light sources 161 may beincluded in the system to direct light onto the object viewing locationand in still other embodiments additional optical elements 162 may beincluded between the light source and the source or object locationposition. In some such variations, the light source itself may becapable of modulated output while in other embodiments any desiredoutput modulation may be achieved by one or more of the optical elements162 or 171-1 such that only at selected times do relevant images fromoptical path 101 actually reach optical paths 102 and 103.

In some embodiments, modulation of the second component optical imagesoccurs by modulating the optical output of the electronic image displaydevice 141 while in other embodiments any necessary modulation isprovided by the optical elements 171-4.

In some embodiment variations, the modulation along the first opticalpath and fourth optical path are coordinated such that only an imagefrom one path or the other reaches that the image capture device at anygiven time. In still other embodiment variations, some overlap ofseparate component optical images is allowed to occur but image captureis controlled so that it occurs only when single component opticalimages are present. In still other embodiment variations no suchcoordination is provided.

In some embodiments, component optical images may be madedistinguishable by wavelength filtering. In other embodiments, componentoptical images may be made distinguishable by optically orelectronically subtracting a known one of the first component optical orsecond component optical images from a composite optical image to yielddata providing an approximation of the other of the component opticalimages. Such captured and extracted component optical image informationmay be used in enhancing composite optical image creation.

In some preferred embodiment variations, component optical imagemodulation, image persistence, and refresh rates are set so as toprovide the eye of an observer with the appearance of a continuouslydisplayed composite optical image while the image capture device cancapture individual component optical images for data manipulationpurposes. In other preferred embodiment variations, the first componentoptical image is presented in such a way so as to provide the appearanceof continuity while the second component optical image is presented soas to provide gaps or discontinuities in the image so that a desireddistinction between first component optical images and second componentoptical images may be achieved by the eye or eyes of the observer. Instill other preferred alternative implementations a reversedpresentation may be provided. In each of these two latter approachesdistinct component optical images may or may not be presented to theimage capture device.

In some variations of the first embodiment, adequate correlation ofimages is achieved by manually adjusting positions, alignment, andfocusing power of one or more optical components and/or by adjusting anoffset parameter or focus plane for the second component optical imageas displayed by the electronic image display device. In furthervariations such manipulations may be performed manually,semi-automatically by electronic or electromechanical devices uponinstruction form an observer or system user while in others they may beperformed in a fully automated manner by the system itself usingcomputer control in combination with feedback information extracted fromsuccessive composite optical images or successive individual componentoptical images that make up the composite optical images that arecaptured by the image capture device and then analyzed by the dataprocessing device.

Data storage, capture, analysis, manipulation, and creation performed bythe data processing device or devices may take on a variety of formsdepending on the details of system usage. For example is the systembeing used for a particular medical procedure, a manufacturinginspection procedure, is the object that is being viewed rigid orflexible, is the object that is being viewed subject to translational orangular movement during the procedure, is the object that is beingviewed subject to changes in shape during the procedure (e.g. subject toresection or ablation, build up by deposition or attachment, expansionor contraction, and the like). Is positioning, size, and orientation ofthe second component optical image to be determined using only theinformation associated with captured first component optical image dataor is it to include feedback information from captured informationassociated with the previous second component optical image informationas well.

In some embodiment variations the composite optical images may becomposed of more than two merged images (e.g. they may be formed fromthree, four, or even more images) with successive component images (e.g.third, fourth or even more component images) being combined withpreviously merged but incomplete composite images.

Second Specific Embodiment

The second specific embodiment, as with the first specific embodiment,provides an augmented reality system 200 and method allowing an observerto view a first component optical image (e.g. a physical optical image)that is combined with an a second component optical image (i.e. anelectronic optical image). In the second specific embodiment a secondbeam splitter is used such that only a physical optical image is capableof reaching the image capture device while a composite optical image iscapable of reaching the eye or eyes of an observer. The details of anexample implementation of this second embodiment can be understood withreference to FIG. 2 wherein like elements are presented with referencenumerals similar to those used in FIG. 1 with the exception that thereference numerals have been updated from the 100 series to the 200series and with other exceptions to be noted below. Due to the presenceof the second beam splitter 211-2 the first component optical image isnot only directed along the third optical path 203 to the image capturedevice 221, it moves from the first beam splitter 211-1 to the secondbeam splitter via the second optical path 202 and then to the eye oreyes of the observer along a fifth optical path 205. The second beamsplitter also projects a portion of the image on the second optical pathalong an unused sixth optical path 206. In this embodiment, the fourthoptical path 204 starting with the electronic image display deviceextends not to the first beam splitter but to the second beam splitterwhere it is split into path 205 and path 206 branches with the path 205branch, in combination with the first component optical image forming acomposite optical image for presentation to the eye or eyes of anobserver.

In this embodiment, as there is no second component optical imagepresented to the image capture device, such information is not availablefor feedback usage in providing enhanced overlaying of first and secondcomponent optical images and thus complete reliance is given to initialoptical set up of the system components and optional system components(e.g. optical components 271-1 to 271-5) and the data analysis and imagedata generation provided by the data manipulation device. In somevariations of this embodiment, the unused composite optical images ofthe sixth optical path 206 may be presented to the eye or eyes of asecond observer, to a display screen, to a second image capture devicefor image storage, for analysis, and/or for providing some form ofsystem feedback. Many of the alternatives noted above for the firstembodiment or set forth in the general discussion may be combined withthe features of this embodiment to create further embodiments orembodiment variations. Conversely some of the features of the thisembodiment, may be combined with those of the first embodiment or otherembodiments presented herein to provide additional system or methodenhancements or simplifications (e.g. intentional creation of additionalimage paths and particularly composite optical image paths allowadditional functionality to be obtained, alternatively simplifiedprocessing may be implemented by not using feedback with some possibleloss in functionality but with faster processing or reduced systemcomplexity and reduced cost)

An advantage of this embodiment over the first embodiment is thattemporal image modulation, or other image separation (e.g. possiblewavelength separation), methods are not needed to ensure that firstcomponent optical image information is available and can be readilyrecognized and used in creating or relative positioning of secondcomponent optical images. Disadvantages of this embodiment involve theinability to use a combination of captured first component optical imagedata and captured second component optical image data to provideenhanced alignment of the component images making up the compositeoptical image.

Third Specific Embodiment

The third embodiment provides an augmented reality system 300 and methodwherein at least two beam splitters are used (e.g. like the secondembodiment) with merging of optical paths occurring by perpendicularpaths of a first component optical image (optical path 301) and a secondcomponent optical image (optical path 304) encountering a first beamsplitter 311-1 wherein the merged optical path/composite optical imageencounters a second beam splitter 311-2 which divides the path into aviewing portion (optical path 305) and a capture portion (optical path303). The details of an example implementation of this third embodimentcan be understood with reference to FIG. 3 wherein like elements arepresented with reference numerals similar to those used in FIGS. 1 and 2with the exception that the reference numerals have been updated fromthe 100 or 200 series to the 300 series and with other exceptions to benoted below.

Many of the alternatives noted above for the first and secondembodiments or set forth in the general discussion may be combined withthe features of this embodiment to create further embodiments orembodiment variations. Conversely the overlaid or composite optical path302 between the beam splitters 311-1 and 311-2 of the present embodimentmay be combined with features of these other embodiments.

A possible advantage of this third embodiment is the creation of anoverlapped optical path 302 (between the two beam splitters) that feedsthe both the paths leading to image capture 303 and viewing 305 whichensures that the presence of an adequately overlaid composite opticalimage presented to the image capture device 321 dictates an adequatelyoverlaid composite optical image being presented to the eye or eyes ofthe observer. A possible disadvantage with this approach relative to thefirst embodiment is that it involves the need for a second beam splitterwhich results in an associated loss in optical image intensity.

Fourth Specific Embodiment

The fourth embodiment (as shown in FIG. 4) provides an augmented realitysystem 400 and method allowing an observer to view a first componentoptical image (e.g. a physical optical image), originating at thebeginning of the first optical path 401, that is combined with a secondcomponent optical image (i.e. an electronic optical image), originatingat the beginning of the fourth optical path 404, along a fifth opticalpath and wherein the overlaying of the two component images occursoptically via three beam splitters from path 404 and via two beamsplitters from path 401 and wherein copies of the first componentoptical images and the second component optical images are sent by thefirst beam splitter 411-1 and a third beam splitter 411-3, respectively,along second and seventh optical paths 402 and 407 into separate imagecapture devices 421-1 and 422-2. In this embodiment, a sixth opticalpath connects one branch of the split image from the third beam splitter411-3 to a second beam splitter 411-2 while the second optical pathconnects one branch of the split image form the first beam splitter to aanother surface of the second beam splitter 411-2. As with the otherembodiments presented herein above, optional optical components 471-1 to471-7 may be located one or more of the first-seventh optical paths. Aswith the second and third embodiments, features of the fourth embodimentare shown using reference numerals similar to those used for FIG. 1 withthe exception that the numerals are now part of the 400 series.

In this embodiment the captured copies of the first component opticalimages and the second component optical images are used, at least inpart, to create, position, and/or align second component optical imagesto be displayed as part of the composite optical image at position 92.An advantage of this embodiment over certain variations of the firstembodiment is that temporal image modulation, or other image separationmethods are not needed to ensure that first component optical imageinformation is made available and can be readily recognized for use increating, positioning or aligning second component optical images withfirst component optical images since such images are captured bydifferent devices. An advantage over the second embodiment is that dueto the presence of the third beam splitter, and second image capturedevice, feedback information is available to allow enhanced firstcomponent optical image and second component optical image alignment toprovide enhanced alignment of the images making up the composite opticalimages. The advantage over the third embodiment is the same as thatcompared to the first embodiment.

Many of the alternatives noted above for the first-third embodiments orset forth in the general discussion may be combined with the features ofthis embodiment to create further embodiments or embodiment variations.Conversely the separate image capture devices of the present embodimentmay be worked into variations of these other embodiments.

Further Alternatives:

Numerous other embodiments are possible that may provide embodimentsthat are distinct form the first four embodiments or may provideenhanced versions of the first to fourth embodiments. Some additionalembodiments may provide a different balance of advantages anddisadvantages relative to the first four embodiments. For example,embodiments using additional beam splitters are possible, embodimentsusing a different arrangement of optical paths are possible, embodimentsthat provide alternative arrangements of viewing paths and image capturepaths are possible. Embodiments using various optical path control andadjustment mechanisms are possible. Such adjustment mechanisms maychange the relative positions and/or orientations of two or more systemcomponents, change the absolute or relative brightness of imagestraveling along the different optical paths, adjust aperture sizes,filter selected wavelengths, provide temporal, wavelength, and/orintensity modulation of images carried by different optical paths. Indifferent embodiments these adjustments may be made manually,semi-automatically, or automatically during a system calibration processor during normal system use, for example, as part of feedbackcorrections. Still other alternative embodiments may add in imagerecording capabilities.

In still other alternative embodiments, electronic images may computercreated images, optically created and then electronically capturedimages, images created by non-light means such as x-rays, CAT scans, NMRscans, ultrasonic scans, or other three-dimensional image generatingtechniques. Such alternative electronic image generating methods may beprovided in a real time manner (e.g. several times per second), in asemi-real-time manner (e.g. several time per minute, one time everyseveral minutes, and the like), or in a static manner. Real time imagesmay be generated from additional image generation source components andimage capture components added to the system while static images may beprovided from imported previously generated data.

Applications:

Numerous specific applications of the methods and systems set forth hereare possible. Two such applications are disclosed in U.S. ProvisionalPatent Application Nos. 61/358,780 and 61/358,793, each filed Jun. 25,2010 and entitled “Ophthalmic Surgical Procedures Using Visual ImagesOverlaid with Visual Representations of Selected Three-Dimensional Data”and “Phacoemulsification Procedures Using Tool Tip to Posterior CapsuleSpacing Information Extracted from Real-Time Diagnostic Scan Data”,respectively. The first of these applications provides enhanced methodsand procedures for placing intra ocular lenses in the eyes of patientswhile the second provide for an enhanced and more controlled methods andprocedures for performing phacoemulsification of the lens of the eyes ofpatients in preparation for placement of IOLs or in preparation forother procedures. The teachings in these referenced applications areincorporated herein by reference as if set forth in full herein.

FURTHER COMMENTS AND CONCLUSIONS

Though various portions of this specification have been provided withheaders, it is not intended that the headers be used to limit theapplication of teachings found in one portion of the specification fromapplying to other portions of the specification. For example, it shouldbe understood that alternatives acknowledged in association with oneembodiment, are intended to apply to all embodiments to the extent thatthe features of the different embodiments make such applicationfunctional and do not otherwise contradict or remove all benefits of theadopted embodiment. Various other embodiments of the present inventionexist. Some of these embodiments may be based on a combination of theteachings herein with various teachings incorporated herein byreference.

In view of the teachings herein, many further embodiments, alternativesin design and uses of the embodiments of the instant invention will beapparent to those of skill in the art. As such, it is not intended thatthe invention be limited to the particular illustrative embodiments,alternatives, and uses described above but instead that it be solelylimited by the claims presented hereafter.

1. A method of providing composite optical images to an eye or eyes ofan observer wherein the composite optical images are comprised of firstcomponent optical images of an object or source along with secondcomponent optical images that are to be displayed in relationship to thefirst component optical images, the method comprising: (a) directingfirst component optical images along a first optical path to a firstbeam splitter; (b) directing the first component optical images alongsecond and third optical paths from the first beam splitter to the eyeor eyes of the observer and to a first image capture device,respectively; (c) producing first electronic data corresponding to thefirst component optical images captured by the first image capturedevice and using said first electronic data in combination with otherdata to provide second electronic data; (d) providing said secondelectronic data to an electronic image display device to provide secondcomponent optical images which are directed along a fourth optical pathonto a second beam splitter; (e) directing the second component opticalimages along at least a fifth optical path that extends from the secondbeam splitter to the eye or eyes of the observer wherein the fifthoptical path and second optical path at least partially overlap.
 2. Themethod of claim 1 wherein the second beam splitter is the same as thefirst beam splitter and the second and fifth optical paths are the sameoptical path.
 3. The method of claim 1 additionally comprising a sixthoptical path extending from the second beam splitter to a second opticalimage capture device and wherein captured second component optical imageinformation provides at least a portion of the other data and isanalyzed in combination with the first electronic data to produce saidsecond electronic data.
 4. The method of claim 1 wherein the secondimage capture device is the same as the first image capture device. 5.The method of claim 4 wherein the first component optical imagescomprise optical images selected from the group consisting of (1)physical optical images, (2) electronic optical images, (3) emissionoptical images, and (4) hybrid optical images.
 6. The method of claim 1wherein the second beam splitter is different from the first beamsplitter.
 7. The method of claim 6 additionally comprising a third beamsplitter that is different from both the first and second beam splittersand which is located along the fourth optical path which splits thefourth optical path into one branch that continues to the second beamsplitter and one branch that directs the second component optical imageonto a second image capture device which in turn provides at leastportion of the other data and is analyzed in combination with the firstelectronic data to produce said second electronic data.
 8. The method ofclaim 5 wherein the first component optical images comprise physicaloptical images and wherein the method additionally comprises directinglight from a light source onto an object located at the beginning of thefirst optical path that gives rise to the physical optical images. 9.The method of claim 8 wherein the light form the light source compriseslight selected from the group consisting of (1) visible light, (2) UVlight, and (3) IR light.
 10. The method of claim 8 additionallycomprising providing one or more optical components between the lightsource and the object wherein at least one of the one or more opticalcomponents is selected from the group consisting of: (1) a lens, (2) amirror, (3) a mechanical or electronic shutter, (4) a prism, (5) adiffraction grating, and (6) a filter.
 11. The method of claim 1additionally comprising providing one or more optical components locatedalong one or more of the first-fifth optical paths wherein the one ormore optical components comprises at least one component selected fromthe group consisting of: (1) a lens, (2) a mirror, (3) a shutter, (4) aprism, (5) a diffraction grating, and (6) a filter.
 12. The method ofclaim 2 wherein the images reaching the beam splitter via the first andfourth optical paths are provided in time modulated forms such thatimages reaching the image capture device along the third optical path atselected times are the images carried along the first optical path butnot the fourth optical path while at other times the images reaching theimage capture device are the images carried along the fourth opticalpath but not the first optical path.
 13. The method of claim 1 whereinthe images represented by the data provided along the second electronicpath and displayed by the electronic image display device do not includeimages provided along the first optical path.
 14. The method of claim 1wherein the using of the first electronic data comprises determiningpositioning information that places the second component optical imagesand the first component optical images within a desired tolerance bycalculating a display position for the second component optical imagesbased, least in part, on captured positions of the first componentoptical images.
 15. The method of claim 14 wherein the calculationfurther comprises spatial comparisons of prior captured first componentoptical images and prior captured second component optical images. 16.The method of claim 1 additionally comprising adjusting an optical pathby adjusting at least one component of the of system, wherein thecomponent is selected from the group consisting of (1) at least one beamsplitter, (2) the display device, (3) at least one image capture device,(4) a positioning of the object, (5) an image focusing component locatedalong one of the optical paths, (7) a filter located along one of theoptical paths, (8) a shutter located along one of the optical paths, (9)an aperture located along one of the optical paths, and (10) a mirrorlocated along one of the optical paths.
 17. The method of claim 16wherein the adjusting step occurs by one or more of: (1) manually duringoperation, (2) manually during calibration, (3) partially automaticallyduring operation, (4) partially automatically during calibration, (5)automatically during operation, and (6) automatically duringcalibration.
 18. The method of claim 17 wherein the adjusting occursautomatically and occurs, at least in part, via feedback provided bysecond component images captured by the second image capture device. 19.The method of claim 1 wherein the using comprises processing the firstelectronic data using a programmed computer that is also programmed tocontrol one or more system components and wherein at least one of theone more system components are selected from: (1) an electronic shutterlocated along one of the optical paths, (2) a light source thatirradiates the object and is capable of output modulation, (3) am imagecapture device, (4) the display device, (5) a focusing device locatedalong one of the optical paths, (6) at least one stage coupled to atleast one component for controlling one or more of a position and anorientation of the component, (7) an electronically controlled aperturelocated along one of the optical paths, and (8) an electronicallycontrolled filter located along at least one of the optical paths. 20.The method of claim 1 wherein (i) the composite optical images areprovided during a procedure selected from the group consisting of (1) amedical procedure, (2) a medical ophthalmic procedure, (3) an ophthalmicintraocular lens replacement procedure, (4) an ophthalmicphacoemulsification procedure, (5) a medical restorative procedure, (6)a medical therapeutic procedure, (7) a medical diagnostic procedure, (8)a medical preventative procedure, (9) a medical research procedure, (10)a manufacturing inspection procedure, (10) a manufacturing assemblyprocedure, and (11) a research or engineering visualization ordemonstration procedure; (ii) the object or source comprises an objectselected from the group consisting of (1) human tissue, (2) a humanorgan, (3) a human eye, (4) a rigid object, (5) a deformable object, (6)an expandable or contractible object and (7) an animate object, (8) aninanimate object; (iii) the second component optical images comprisesone or more images selected from the group consisting of (1) capturedphysical object images, (2) magnetic resonance images, (3) coherencetomography image(s), (4) optical coherence tomography images, (5) acomputer generated polygonal graphic representation of an object, (6) animage that is in a format that can be manipulated by a graphicalanimation program, (7) an image that is in a format the can bemanipulated by a graphical 3-D CAD program.
 21. A method of providingcomposite optical images to an eye or eyes of an observer wherein thecomposite optical images are comprised of first component optical imagesof a source or object along with second component optical images thatare to be displayed in relationship to the first component opticalimages, the method comprising: (a) directing first component opticalimages from a source or object to a first beam splitter and thereafterdirecting split first component optical images to a first image capturedevice located along a first image capture path and to a viewinglocation located along a first viewing location path; (b) directingsecond component optical images from an electronic image display deviceto a second beam splitter and thereafter directing split secondcomponent optical images to a viewing location along a second viewinglocation path having at least a terminal portion that overlays aterminal portion of the first viewing location path; (c) using at aportion of data created from the first component optical image capturedby the first image capture device in generating electronic image datathat is provided to an electronic image display device to create updatedsecond component optical images.
 22. The method of claim 21 wherein thesecond beam splitter is the same as the first beam splitter.
 23. Themethod of claim 21 wherein the first component optical images compriseand optical image selected from the group comprising (1) physicaloptical images, (2) electronic optical images, (3) emission opticalimages, and (4) hybrid optical images.
 24. The method of claim 21wherein the second beam splitter is different from the first beamsplitter.
 25. The method of claim 21 wherein the first component opticalimages comprise physical optical images and wherein the methodadditionally comprises directing light from a light source onto anobject located at the beginning of the first optical path that givesrise to the physical optical images.
 26. The method of claim 25 whereinthe light form the light source comprises light selected from the groupconsisting of (1) visible light, (2) UV light, and (3) IR light.
 27. Themethod of claim 26 additionally comprising providing one or more opticalcomponents between the light source and the object wherein at least oneof the one or more optical components is selected from the groupconsisting of: (1) a lens, (2) a mirror, (3) a mechanical or electronicshutter, (4) a prism, (5) a diffraction grating, and (6) a filter. 28.The method of claim 27 additionally comprising providing one or moreoptical components located along a path traveled by either one or bothof the first component optical image and the second component opticalimage, wherein the one or more optical components comprises at least onecomponent selected from the group consisting of: (1) a lens, (2) amirror, (3) a shutter, (4) a prism, (5) a diffraction grating, and (6) afilter.
 29. The method of claim 26 wherein at least one of the firstcomponent optical image or second component optical image reaching thebeam splitter are provided in time modulated forms such that imagesreaching an image capture device at a given times are only firstcomponent optical images while at other times are the images reaching animage capture device are only second component optical images.
 30. Themethod of 26 wherein the second component optical images do not compriseimage that are the same as first component optical images.